The rise of artificial intelligence and data science across industries underscores the pressing need for effective management and governance of machine learning (ML) models. Traditional approaches to ML models management often involve disparate storage systems and lack standardized methodologies for versioning, audit, and re-use. Inspired by data lake concepts, this paper develops the concept of ML Model Lake as a centralized management framework for datasets, codes, and models within organizations environments. We provide an in-depth exploration of the Model Lake concept, delineating its architectural foundations, key components, operational benefits, and practical challenges. We discuss the transformative potential of adopting a Model Lake approach, such as enhanced model lifecycle management, discovery, audit, and reusabil-ity. Furthermore, we illustrate a real-world application of Model Lake and its transformative impact on data, code and model management practices.

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.

Model serving and deployment is one of the pillars of the MLOps stack. In this article, I'll dive into it and talk about what a basic, intermediate, and advanced setup for model serving look like. Let's start by covering some basics. Training a machine learning model may seem like a great accomplishment, but in practice, it's not even halfway from delivering business value. For a machine learning initiative to succeed, we need to deploy that model and ensure it meets our performance and reliability requirements. You may say, "But I can just pack it into a Docker image and be done with it". In some scenarios, that could indeed be enough. When people talk about productionizing ML models, they use the term serving rather than simply deployment. So what does this mean?

In speaking with many machine learning teams, we've found that implementing a model registry has become a priority for AI-first organizations in solving visibility and governance concerns. A model registry is a centralized model store to collaboratively manage the full lifecycle of ML models. This includes model lineage and versioning, moving models between stages from development to staging to production, and model annotations and discovery (i.e., timestamps, descriptions, labels, etc.). ML teams implement a model registry solution to get centralized visibility and management of their models. But there are challenges to adopting a model registry, making it hard to build an up-to-date model registry that contains everything an organization needs.

MLOps has quickly become one of the most important components of data science, with the market expected to grow by almost $4 billion by 2025. It is already being leveraged heavily with companies like Amazon, Google, Microsoft, IBM, H2O, Domino, DataRobot and Grid.ai using MLOps for pipeline automation, monitoring, lifecycle management and governance. More and more MLOps tools are being developed to address different parts of the workflow, with two dominating the space, Kubeflow and MLflow. Given their open-sourced nature, Kubeflow and MLflow are both chosen by leading tech companies. However, their capabilities and offerings are quite different when compared. For example, while Kubeflow is pipeline focused, MLflow is experimentation based.

Iterative, the MLOps company dedicated to streamlining the workflow of data scientists and machine learning (ML) engineers, has launched machine learning engineering management (MLEM) - an open source model deployment and registry tool that uses an organisation's existing Git infrastructure and workflows. According to the company, MLEM is designed to bridge the gap between ML engineers and DevOps teams. DevOps teams can understand the underlying frameworks and libraries a model uses and automate deployment into a one-step process for production services and apps, Iterative states. IDC AI/ML Lifecycle Management Softwrae research director Sriram Subramanian says, "Iterative enables customers to treat AI models as just another type of software artifact. The ability to build ML model registries using Git infrastructure and DevOps principles allows models to get into production faster."

MLOps platform Iterative, which announced a $20 million Series A round almost exactly a year ago, today launched MLEM, an open-source git-based machine learning model management and deployment tool. The idea here, the company says, is to bridge the gap between ML engineers and DevOps teams by using the git-based approach that developers are already familiar with. Using MLEM, developers can store and track their ML models throughout their lifecycle. As such, it complements Iterative's open-source GTO artifact registry and DVC, the company's version control system for data and models. "Having a machine learning model registry is becoming an essential part of the machine learning technology stack. Current SaaS solutions can lead to a divergence in the lifecycle of ML models and software applications," said Dmitry Petrov, co-founder and CEO of Iterative.

Experiment tracking has been one of the most popular topics in the context of machine learning projects. It is difficult to imagine a new project being developed without tracking each experiment's run history, parameters, and metrics. While some projects may use more "primitive" solutions like storing all the experiment metadata in spreadsheets, it is definitely not a good practice. It will become really tedious as soon as the team grows and schedules more and more experiments. Many mature and actively developed tools can help your team track machine learning experiments. In this article, I will introduce and describe some of these tools, including TensorBoard, MLFlow, and Neptune.ai,

For any business, seamless deployment of ML models into production is the key to success of its live analytics use cases. In this article, we will learn about deploying ML models on AWS (Amazon Web Services) using MLflow and also look at different ways to productionize them. Subsequently, we will explore the same process on the two other popular platforms: Azure and GCP. An Identity and Access Management execution role defined that grants SageMaker access to the S3 buckets. Once the above steps are done with, here's how we proceed with the deployment process on AWS - Before any model can actually be deployed on SageMaker, Amazon workspace needs to be set up.

Machine learning operations (MLOps) are key to effectively transition from an experimentation phase to production. The practice provides you the ability to create a repeatable mechanism to build, train, deploy, and manage machine learning models. To quickly adopt MLOps, you often require capabilities that use your existing toolsets and expertise. Projects in Amazon SageMaker give organizations the ability to easily set up and standardize developer environments for data scientists and CI/CD (continuous integration, continuous delivery) systems for MLOps engineers. With SageMaker projects, MLOps engineers or organization administrators can define templates that bootstrap the ML workflow with source version control, automated ML pipelines, and a set of code to quickly start iterating over ML use cases.